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Abstract 

Cyberbullying is a growing problem affecting more than 
half of all American teens. The main goal of this paper 
is to investigate fundamentally new approaches to un¬ 
derstand and automatically detect incidents of cyberbul¬ 
lying over images in Instagram, a media-based mobile 
social network. To this end, we have collected a sam¬ 
ple Instagram data set consisting of images and their 
associated comments, and designed a labeling study for 
cyberbullying as well as image content using human la¬ 
belers at the crowd-sourced Crowdflower Web site. An 
analysis of the labeled data is then presented, including 
a study of correlations between different features and 
cyberbullying as well as cyberaggression. Using the la¬ 
beled data, we further design and evaluate the accuracy 
of a classifier to automatically detect incidents of cyber¬ 
bullying. 

Introduction 

As online social networks (OSNs) have grown in popularity, 
instances of cyberbullying in OSNs have become an increas¬ 
ing concern. In fact more than half of American teens have 
been the victims of cyberbullying (National Crime Preven¬ 
tion Council 2011). Although cyberbullying may not cause 
any physical damage initially, it has potentially devastating 
psychological effects like depression, low self-esteem, sui¬ 
cide ideation, and even suicide (Hinduja and Patchin 2010; 
E. Menesini 2009). Incidents of cyberbullying with extreme 
consequences such as suicide are now routinely reported in 
the popular press. For example, Phoebe Prince, a 15-year- 
old high school girl, committed suicide after being cyberbul- 
lied by negative comments in the Facebook social network 
(Goldman 2010). Hannah Smith, a 14-year-old, hanged her¬ 
self after negative comments were posted on her Ask.fm 
page, a popular social network among teenagers (Smith- 
Spark 2013). Cyberbullying is such a serious problem that 
nine suicides have been linked with cyberbullying on the 
Ask.fm Web site alone (Broderick 2013). Although cyber¬ 
bullying is not the direct cause of these suicides, cyberbully¬ 
ing was viewed as a contributing factor in the death of these 
teenagers (Cyberbullying Research Center 2013). 
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Given the gravity of the consequences cyberbullying has 
on its victims and its rapid spread among middle and high 
school students, there is an immediate and pressing need for 
research to understand how cyberbullying occurs in OSNs 
today, so that effective techniques can be developed to ac¬ 
curately detect cyberbullying. A recent survey on cyber¬ 
bullying (Ditch the Fabel Anti Bullying Charity 2013) has 
listed the top five networks where the highest percentage 
of users have reported experiencing cyberbullying, namely 
Facebook, Twitter, YouTube, Ask.fm, and Instagram. Insta¬ 
gram is of particular interest as it is a media-based mobile 
social network, which allows users to post and comment on 
images. Cyberbullying in Instagram can happen in differ¬ 
ent ways, including posting a humiliating image of someone 
else by perhaps editing the image, posting mean or hateful 
comments, aggressive captions or hashtags, or creating fake 
profiles pretending to be someone else (Silva et al. 2013). 
Figure 1 illustrates an example of an attack in Instagram in 
which offensive hashtags and hateful comments were posted 
to humiliate the profile owner. 

Cyberbullying has been defined as intentionally aggres¬ 
sive behavior that is repeatedly carried out in an online con¬ 
text against a person who cannot easily defend him or herself 
(Kowalski et al. 2012; Patchin and Hinduja 2012). It is im¬ 
portant to this definition of cyberbullying that both the fre¬ 
quency of negativity and the imbalance of power between 
the victim and perpetrator be taken into account. In contrast, 
cyberaggression is a more general type of behavior that is 
broadly defined as using digital media to intentionally harm 
another person (Kowalski et al. 2012). 

Prior works that investigated cyberbullying (Ptaszynski et 
al. 2010),(Dadvar et al. 2012),(K. Reynolds and Edwards 
201 l),(Dinakaret al. 2012),(H. Sanchez 2012),(Kontostathis 
et al. 2013), (Xu et al. 2012),(Nahar et al. 2014),(Na- 
har, Fi, and Pang 2013),(Dinakar, Reichart, and Fieberman 
2011),(Nahar et al. 2012) are more accurately described as 
research on cyberaggression, since these works do not take 
into account both the frequency of negativity and the im¬ 
balance of power. These works have largely applied a text 
analysis approach to online comments, since this approach 
results in higher precision and lower false positives than sim¬ 
pler list-based matching of negative words (Sood, Antin, and 
Churchill 2012). Other work analyzed negativity in Ask.fm 
(Hosseinmardi et al. 2014a) and Instagram comments (Hos- 



seinmardi et al. 2014b), but did not label the data at all. 
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Figure 1: An example of comments posted on Instagram. To 
give more room for the text, we have moved the associated 
image to overlay some of the text. 

Additional research investigated aspects of the Instagram 
social network, but not in the context of cyberbullying. For 
example, (Weilenmann, Hillman, and Jungselius 2013) ex¬ 
plored users’ photo sharing experience in a museum. (Silva 
et al. 2013) considered the temporal photo sharing behav¬ 
ior of Instagram users. (Y. Hu 2014) categorized Instagram 
images into eight popular image categories and the Insta¬ 
gram users into five types in terms of their posted images. 
By investigating user practices in Instagram, (Araujo et al. 
2014) concluded that users tend to be more active during 
weekends and at the end of the day. They also found out that 
users are more likely to like and comment on the medias 
that are already popular, thereby inducing the rich get richer 
phenomenon. 

In this paper, we make the following contributions: 

• We provide a clear distinction between cyberbullying and 
general cyberaggression. Cyberbullying is one type of cy¬ 
beraggression, and most of the earlier research in this area 
has focused on identifying cyberagression, which is rela¬ 
tively easier than identifying cyberbullying. 

• We investigate cyberbullying behavior in Instagram by 
labeling collected Instagram images and their associated 
discussion comments according to both the more restric¬ 
tive definition of cyberbullying and the more general def¬ 
inition of cyberaggression. 

• We present an analysis of the labeled images and com¬ 
ments, including the relationships of cyberbullying and 
cyberaggression to a variety of features, such as number 
of associated comments, N-grams, followed-by and fol¬ 
lowing behavior of the posting users, liking behavior, fre¬ 
quency of comments, and labeled image content. 


• We design and evaluate multi-modal classifiers to detect 
cyberbullying based on the labeled data, measuring accu¬ 
racy across different feature sets including text, images, 
and meta data. 


Data Collection 

Using a snowball sampling method, we have identified 41K 
Instagram user ids. 61% of these Instagram ids have pub¬ 
lic profiles, which is about 25K public profiles. These 25K 
public user profiles comprise our complete set of typical In¬ 
stagram users data. For each public Instagram user, the col¬ 
lected profile data includes the media objects/images that the 
user has posted and their last 150 associated comments, user 
id of each user followed by this user, user id of each user 
who follows this user, and user id of each user who com¬ 
mented on or liked the media objects shared by the user. We 
consider each media object/image and its associated com¬ 
ments as a media session. For this set of 25K users, 697K 
media sessions were collected. 

In order to make the labeling of cyberbullying more man¬ 
ageable, we sought to label a smaller subset of these media 
sessions. We focused on those media sessions that have a 
high percentage of negativity in their associated comments, 
since we reasoned that this should give us a higher likelihood 
of identifying cyberbullying once the data was properly la¬ 
beled. We used the same approach as in the previous works 
(Hosseinmardi et al. 2014a; 2014b) for tagging a com¬ 
ments as a negative or not, by looking for profanity words 
coming from a dictionary obtain form (NoSwearing.com ; 
von Ahn’s Research Group 2014). Specifically, we select im¬ 
ages using the following two criteria: 

• the media has at least 15 comments, and 

• more than 40% of the comments by users other than the 

profile owner have at least one negative word. 

Using these criteria, we were able to reduce the number 
of media sessions to a more tractable group of about 1, 203. 
When we returned to Instagram to collect images associated 
to the comments of selected media’s for labeling, only 998 
were still available, for the rest either the media session was 
deleted or the profile was made private or deleted. The rea¬ 
son for putting lower bound on the number of comments 
(minimum 15 comments) is to ensure that there are enough 
comments to adequately assess the frequency or repetition 
of negativity, which is an important part of the cyberbully¬ 
ing definition. The average number of comments per image 
is 59.6, and we decided to analyze all images with number 
of comments at least a quarter of this average number. For 
these 998 media sessions, the average number of comments 
associated with a media is about 64.3. 

Figure 2 shows the distribution of the number of com¬ 
ments for our selected smaller subset of media sessions com¬ 
pared with the number of comments for the complete set 
of media sessions. We observe that the fraction of images 
with number of comments between 15 and 50 is higher in 
the selected data set than that in the complete set. However, 
the distribution is similar when the number of comments is 
greater than 50. This shows that media sessions with rel- 











atively higher negativity tend to be confined to moderate 
number of comments. 



Figure 2: Comparison of the distribution of the number of 
comments per collected Instagram media session. Blue is for 
the complete set of media sessions, and red is for the selected 
subset of 998 media sessions with more than 15 comments 
and high degree of negativity. 


Figure 3 illustrates the CCDF of the number of followed 
by and follows for users in both the complete and selected 
set of media sessions. We see that the number of follows 
for users in the complete and selected sets exhibit the same 
pattern. However, the distribution for selected users ends at 
around 7,500, while the distribution for all users goes to al¬ 
most 10'. On the other hand, distributions of the number 
of followed by users are different for selected users and all 
users. The number of followed by users is higher for the se¬ 
lected users, but this distribution ends at around 4.16 * 10 6 , 
while the distribution for all users goes all the way up to 10 8 . 
Looking at the data more closely, we found that a large num¬ 
ber of images posted by the selected users set correspond to 
popular personalities or events, e.g., a lot of these users are 
singers, celebrities, tattoo artists, or simply users who are 
popular within a local area. These users draw a lot of at¬ 
tention. Because of their popularity, they have a relatively 
larger number of followers, and tend to attract a significant 
number of negative comments in the form of criticism from 
other users. 



Cyberbullying Labeling 

In this section, we explain the design and methodology for 
our survey labeling the selected set of media sessions. Our 
first challenge is choosing appropriate definitions of terms, 
which will then be used in ground truth labeling. Based on 
the literature, a major early choice that we have made is to 
distinguish between cyberaggression and cyberbullying. Cy¬ 
beraggression is broadly defined as using digital media to 
intentionally harm another person (Kowalski et al. 2012) . 
Examples include negative content and words such as pro¬ 
fanity, slang and abbreviations that would be used in nega¬ 
tive posts such as hate, fight, wtf. Cyberbullying is one form 
of cyberaggression that is more restrictively defined as in¬ 
tentional aggression that is repeatedly carried out in an elec¬ 
tronic context against a person who cannot easily defend him 
or herself (Kowalski et al. 2012; Patchin and Hinduja 2012). 
Thus, cyberbullying consists of three main features : (1) an 
act of aggression online; (2) an imbalance of power between 
the individuals involved; and (3) it is repeated over time 
(Hunter, Boyle, and Warden 2007; Kowalski et al. 2012; 
Olweus 1993; 2013; Smith, del Barrio, and Tokunaga 2012). 
The power imbalance can take on a variety of forms in¬ 
cluding physical, social, relational or psychological (Doo¬ 
ley, Pyzalski, and Cross 2009; Monks and Smith 2006; 
Olweus 2013; Pyzalski 2010), such as a user being more 
technologically savvy than another (Kowalski et al. 2014), a 
group of users targeting one user, or a popular user targeting 
a less popular one (Limber, Kowalski, and Agatston 2008). 
Repetition of cyberbullying can occur over time or by for¬ 
warding/sharing a negative comment or photo with multiple 
individuals (Limber, Kowalski, and Agatston 2008). 
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Is there any cyberaggressive behavior in the online posts? Mark yes if there is at least one 
negative word/comment and or content with intent to harm someone or others. 

O No 
O Yes 

Is there any cyerbullying in the online post? Mark yes if there are negative words and or 
comment with intent to harm someone or other, and the posts include two or more 
repeated negativity against a victim that cannot easily defend him or herself 

O No 
O Yes 


Figure 4: An example of the labeling survey, which shows 
an image and its corresponding comments, and the survey 
questions. 


Figure 3: CCDF of the number of followed by and follows 
for users in the complete set and highly negative subset of 
media sessions. 


In Instagram, each media session consists of a media 
posted by the profile owner and the corresponding comments 
for the media object. The goal in this paper is to investigate 
cyberaggression and cyberbullying in this multi-modal (tex- 
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Figure 5: Fraction of media sessions that have been voted k times as cyberagression (left) or cyberbullying (right). 


tual comments and media objects) context. Therefore, the 
design of our survey needed to incorporate both the image 
and the associated text comments when asking the human 
labeler whether the media session was an instance of cyber¬ 
bullying or cyberaggression. Figure 4 illustrates an example 
of our design for the labeling survey. On the left is the image, 
and on the right is a scrollable interface to help the labeler 
see all of the comments associated with this image. With the 
help of an expert, we decided to ask the labelers two ques¬ 
tions, namely whether the media session constituted cyber¬ 
aggression or not, and whether the media session constituted 
cyberbullying or not. During the instructional phase prior to 
labeling, labelers were given the aforementioned definitions 
of cyberaggression and cyberbullying along with related ex¬ 
amples. Each media session was labeled by five contributors. 


To monitor the quality of labeling, potential contributors 
were given the answers to a set of examples, and then were 
subjected to a pre-filtering step in which they were asked to 
answer a set of similar quiz questions. Contributors needed 
to answer correctly a minimum number of quiz questions 
to qualify as a labeler for our survey. Also during the job, 
random test questions were asked to monitor the quality of 
the labeling during the job. A minimum threshold amount 
of time was also set to filter out contributors who rushed 
through the labeling without spending a sufficient minimum 
duration to ensure the quality of the labeling. 


We were also interested in image contents of media ses¬ 
sions that had been targeted with a high proportion of neg¬ 
ative comments. If the type or category of an image could 
be identified, then this may prove to be a useful feature in 
classification of cyberbullying. We first sampled some of the 
images in the selected subset to determine a suitable set of 
representative categories or types to be used in the label¬ 
ing. For example, some of the dominant categories were the 
presence of a human in the image, as well as animals, text, 
clothes, tattoos, sports and celebrities. We then conducted a 
second survey focused only on the image content, and asked 
labelers to identify which of the aforementioned categories 
were present in the image. Multiple categories could be se¬ 
lected for a given image. 


Analysis and Characterization of Labeled 
Cyberbullying 

We submitted our survey with 998 media sessions (images 
and their associated comments) to CrowdFlower, a crowd 
sourcing website, each labeled by five different contributors. 
Figure 5 illustrates the distribution of the labeled answers 
among the five labelers for each of the two questions on cy¬ 
berbullying and cyberaggression. The higher the number of 
votes for a given label, the more confidence that we have that 
a given media session constitutes cyberaggression or cyber¬ 
bullying, where five votes constitutes unanimous agreement. 
The left chart in Figure 5 shows the percentage of samples 
that have been labeled as cyberaggression k times, and the 
right chart shows the percentage of samples that have been 
labeled as cyberbullying k times. 

We notice that for cyberaggression, most of the probabil¬ 
ity mass is around media sessions labeled as cyberaggres¬ 
sion by all five contributors. This is not surprising since all 
the samples have at least 40% negative comments. However, 
we observe that around 17% of the media sessions have not 
been labeled as cyberaggression at all by any of the five con¬ 
tributors. This suggests that only employing a high percent¬ 
age of negativity threshold of 40% to detect cyberaggression 
can still produce many false alarms. 

For cyberbullying labeling (right chart in Figure 5), we 
notice that about 24% of the media sessions have not been 
labeled as cyberbullying by any of the five contributors, even 
though these samples were originally selected for their high 
negativity. Further, we observe that about 48% of the me¬ 
dia sessions have two or fewer votes. If we apply a majority 
vote criterion to deciding whether a given session was cyber¬ 
bullying or not among the five labelers, then nearly half of 
the sessions would be defined as not cyberbullying, despite 
their high percentage negativity. Therefore, a key finding of 
our labeling is that a large fraction of Instag ram media ses¬ 
sions with a high percentage of negative words would not be 
deemed as cyberbullying. The implication is that classifier 
design for cyberbullying here cannot solely rely on the per¬ 
centage of negativity among the words in the image-based 
discussion, since this would produce many false positives, 
but instead must consider other features to improve accu¬ 
racy. 

Another key observation is that the labelers are mostly 
in agreement about what behavior constitutes cyberbully- 
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cyberbullying 

0.069 

0.04 
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-0.02 

cyberaggression 

0.04 

0.07 

0.14 

-0.00 


Table 1: Correlation between number of votes for cyberbullying /cyberagression and image/user meta data 


ing and what does not in Instagram media sessions. That is, 
most of the labelers agree either that a media session is cy¬ 
berbullying (about 40% of sessions have either four or all 
five votes for cyberbullying) or that it is not (about 38% 
of sessions have either zero or one vote). Only about 22% 
of media sessions have two to three votes, so there is some 
disagreement in a small fraction of cases about whether the 
session is cyberbullying or not. 



Bullying 


Figure 6: Two-dimensional distribution of media sessions as 
a function of the number of votes given for cyberagression 
versus the number of votes given for cyberbullying, assum¬ 
ing five labelers. 

In order to understand the relationship between labeled 
cyberaggression and labeled cyberbullying media sessions, 
we plotted in Figure 6 a two-dimensional heat map that 
shows the distribution of media sessions as a function of 
the number of votes each media session received for cyber¬ 
aggression and cyberbullying. We observe that a significant 
fraction of the sessions exhibit strong agreement in terms of 
both receiving high numbers of votes for both cyberbully¬ 
ing and cyberaggressions, or both receiving low numbers of 
votes, i.e. the session is neither cyberbullying nor cyberag¬ 
gression. This is shown by the high energy in the upper right 
and lower left along the diagonal. In addition, it is promising 
that the area below the diagonal is essentially zero, meaning 
no sessions have received more votes for cyberbullying than 
cyberaggression. This conforms with the definition that cy¬ 
berbullying is a subset of cyberaggression. 

We see that the remaining significant energy in the dis¬ 
tribution appears in the area above the diagonal. Media ses¬ 
sions in this area exhibit the property that if they receive Ni 
cyberbullying labels, then they receive > Ni cyberag¬ 
gression labels. This area corresponds to cases where there 
is cyberaggression but not cyberbullying. In particular, if we 
look at the cases where there is some disagreement as to 
whether a session is cyberbullying or not (A'| = 2 or 3 votes 
for cyberbullying), we see that there is significant support 


that these sessions exhibit cyberaggression (there is signifi¬ 
cant energy for N 2 values of four and five votes for cyber¬ 
aggression). In fact, the dominant value for cyberagression 
when Ni = 2 is N 2 = 4, and similarly the dominant value 
for cyberaggression for N\ = 3 is iV2 = 4 or 5. As a re¬ 
sult, our analysis is able to quantify that there is substantial 
support for identifying Instagram media sessions that exhibit 
cyberaggression but not cyberbullying. 

Next, we would like to examine the correlation between 
the strength of labeled cyberbullying/cyberaggression and a 
variety of other factors. We define the strength of cyberbul¬ 
lying as the number of votes received for labeling a media 
session as cyberbullying, and similarly for cyberaggression. 
Table 1 shows the correlation between the strength of cy¬ 
berbullying/cyberaggression and media properties such as 
the number of likes, as well as meta data about the profile 
owner of the shared media object, such as the number of 
followings, followed-by’s, and total shared media. We ob¬ 
serve that there is a correlation of 0.17 with the number of 
followed-by’s while there is no significant correlation with 
the number of likes, total shared media, and followings. We 
also found that cyberaggression exhibits similar but slightly 
weaker correlations to the same factors. 



Figure 7: Temporal correlation analysis. 


Our analysis further explores Pearson’s correlation by 
considering temporal factors. We would like to understand 
how the human labelers incorporated the definition of cyber¬ 
bullying, which includes the temporal notion of repetition of 
negativity over time, into their labeling. Given time stamps 
on every collected comment, we compute the interval or in¬ 
terarrival time between a comment and the next comment. 
We then counted the number of comment interarrival times 
in a media session less than some threshold value. Figure 7 
describes our results, namely that there is a strong corre¬ 
lation of about 0.4 between the strength of support for cy¬ 
berbullying and media sessions in which there are frequent 
postings within 1 hour of each other. Further, we find that as 
we expand the allowable duration between comments, that 
is comments are allowed to be further apart in time, then 
the correlation weakens considerably between more widely 



















separated comments and support for labeling this session as 
cyberbullying. We also considered cyberbullying’s correla¬ 
tion with other temporal factors such as the median, mean 
and variance, i.e. jitter, of the comment interarrival times 
but found little correlation. Cyberaggression temporal cor¬ 
relations follow a similar pattern. 

To summarize, we have found that there are strong corre¬ 
lations between the strength of support for labeled cyberbul¬ 
lying and the number of text comments as well as the tem¬ 
poral property of the number of comments that are posted 
within one hour of one another in an Instag ram media ses¬ 
sion. 

Image Labeling Analysis 

In this subsection, we would like to understand the rela¬ 
tionship between image content and cyberbullying in a me¬ 
dia session. Towards this end, we display the distribution 
results of our second survey on labeling image content in 
Figures 8 and 9. First, we observe that among the media 
sessions with the highest negativity, the most common la¬ 
bels for the image content in these media sessions are Per¬ 
son/People, Text, Sports, and perhaps Tattoo, for most values 
of support for cyberbullying. Second, there is some skew in 
distributions for certain labels such as Person/People, Tat¬ 
too and Sports, as the amount of support for cyberbullying 
varies. For example, for images labeled as containing a Tat¬ 
too, we see a strong skew towards lower values of cyberbul¬ 
lying. Such a skew may be helpful in classifier design, since 
whenever a tattoo is present, there appears to be little sup¬ 
port that there is cyberbullying occurring, while whenever 
there is strong support for cyberbullying, images with tat¬ 
toos are more scarce. For Person/People, we see a skew in 
the opposite direction towards more cyberbullying support, 
and similarly for Sports. Similar behavior is exhibited for 
cyberaggression as well. 

Since labeling of image content into more than one cat¬ 
egory was permitted, then we are further interested to see 
the distribution of multi-label images. Figure 10 shows the 
fraction of other categories assigned to a Person/People la¬ 
beled image. For example. Figure 10 shows that more than 
60% of images labeled with Person/People were exclusively 
labeled as such, but about 15% of such images were also la¬ 
beled with the Text label. Very few images were labeled with 
three labels. 


Figure 10: Fraction of other categories assigned to an image 
given that the image has been labeled as Person/People. 


Classifier Design and Evaluation 

To design and train the classifier, we chose to apply a major¬ 
ity vote criterion on the labeled data to determine whether 
a media session was cyberbullying or not. Further, Crowd- 
Flower provides us with a degree of trust for each labeler 
based on the percentage of correctly answered quiz and test 
questions during the labeling session. This trust value is 
incorporated by CrowdFlower into a weighted version of 
the majority voting method called a “confidence level”. We 
decided to employ this weighted trust-based majority vot¬ 
ing metric as the basis for our classifier design. Media ses¬ 
sions whose weighted trust-based metric was equal to or 
greater than 60% were deemed to be strong enough sup¬ 
port for cyberbullying. Actually, 90% of the original pure 
majority-vote based media sessions wound up in this higher- 
confidence cyberbullying-labeled group. For this higher- 
confidence data set, 52% in total belonged to the “bullying” 
group while 48% were not deemed to be bullying. This pro¬ 
vides a base case from which to compare our classifier since 
we can simply apply a detector based on the 40% negativity 
threshold and achieve 0.52 accuracy for cyberbullying de¬ 
tection. 

Two types of features were evaluated, namely those fea¬ 
tures obtained from the content of comments, and those fea¬ 
tures obtained from shared media objects and the profile 
owner. For the text features, first we applied a pre-processing 
step to remove characters such as “!”, etc. and stop 
words such as “and”,“or”,“for”, etc. Features extracted from 
text include unigram, bigram, 3-gram,number of comments 
for the image, and number of posts within interval less than 
one hour. Features extracted from user and media informa¬ 
tion (named as meta data) includes the number of followed- 
by’s, follows, likes, and shared medias and features extracted 
from image content includes image categories. 

Table 2 illustrates the best performance results among dif¬ 
ferent examined classifiers (all numbers are average over 10- 
fold cross validation results). In the first row using low di¬ 
mensional feature space of meta data and a simple Naive 
Bayes Classifier we jumped to accuracy 0.71 from base¬ 
line 0.52. Next we observed that adding image categories 
increased the accuracy to 0.72, with a high recall 0.78. 

In another experiment, only the text features unigram and 
3-gram gave us the best accuracy using linear Support Vector 
Machine (SVM) Classifier. However, the dimension of uni¬ 
grams and 3-gram features is very high, so next row shows 
the accuracy after applying Singular Value Decomposition 
(SVD) on text features. We observed keeping only the first 
200 components, we can get the same accuracy. 

In the next step we added meta data and image categories 
to the text features. To get the best accuracy, we first stan¬ 
dardized these set of features, applied kernel PCA (Princi¬ 
ple Component Analysis) and kept the first 20 components. 
Then we concatenate this set of reduced dimension features 
with the reduced dimension features obtained from text. Ap¬ 
plying linear SVM classifier, the accuracy jumped to 0.87 
with both high precision and recall. 

In summary, by employing multi-modal features obtained 
from text, meta data and images as input into a linear SVM 
classifier, the accuracy of cyberbullying detection was mean- 
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Figure 8: Distribution of image categories given the media sessions have been voted for k times for cyberbullying. As some 
images belong to more than one category, the bars will sum up to more than one. 
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Figure 9: Distribution of image categories given the media sessions have been voted for k times for cyberaggression. 


ingfully improved by 0.35 to a total of 0.87 compared to 
a base case of 0.52. Simple meta data features gain accu¬ 
racy 0.71, but to increase recall, more complex features are 
needed. 

Discussion and Future Work 

One theme for future work is to improve the performance 
of our classifier by adding more input features, such as new 
image features, temporal behavior of commenting, mobile 
sensor data, etc. A limitation of our current classifier is that 
it is designed only for highly negative media sessions. A 
more general classifier that can apply to all media sessions 
is needed. This will also require us to enlarge our labeled 
data set substantially. Incorporating image features needs to 
be automated by applying image recognition algorithms. We 
plan to explore this research direction as well. We have ap¬ 
plied a majority vote definition in designing our classifier. 
Another definition to consider is when at least one labeler 
has declared that he/she thinks this media session constitutes 
cyberbullying. New classifiers will have to be designed for 
this definition. 

We also plan to consider designing classifiers for cyber¬ 
aggression in addition to cyberbullying, and to investigate 
those media sessions that represent the former but not the 
latter behavior. 

Another theme for future work is to obtain greater detail 
from the labeling surveys. Our experience was that stream¬ 
lining the survey improved the response rate, quality and 
speed. However, we desire more detailed labeling, such as 
for different roles in cyberbullying - identifying and differ¬ 
entiating the role of a victim’s defender, who may also spew 


negativity, from a victim’s bully or bullies. 

Conclusions 

We believe this paper makes the following major contribu¬ 
tions: an appropriate definition of cyberbullying that incor¬ 
porates both frequency of negativity and imbalance power 
is applied in large-scale labeling, and is differentiated from 
cyberaggression; cyberbullying is studied in the context of a 
media-based social network, incorporating both images and 
comments in the labeling; a detailed analysis of the distri¬ 
bution results of the labeling of cyberbullying incidents is 
presented, including a correlation analysis of cyberbullying 
with other factors derived from images, text comments, and 
social network meta data; multi-modal classification results 
are presented that incorporate a variety of features to iden¬ 
tify cyberbullying incidents. 

The major findings of this paper comprise the following 
results. First, a key finding of our labeling is that about 48% 
of Instagram media sessions were not deemed as cyberbully¬ 
ing using a majority vote criterion among five labelers, even 
though these were among the media sessions with the high¬ 
est percentage of profanity words, i.e. a significant fraction 
of negative content does not constitute acts of online cy¬ 
berbullying. Second, labelers are mostly in agreement about 
what behavior constitutes cyberbullying and what does not 
in Instagram media sessions. Third, our analysis identified 
that that there is significant class of Instagram media ses¬ 
sions that exhibits cyberaggression but not cyberbullying. 
Fourth, there are strong correlations between the strength 
of support for labeled cyberbullying and the number of text 
comments as well as the temporal property of the number 





































Features 

Classifier 

Accuracy 

Precision 

Recall 

Meta data 

Naive Bayes 

0.71 

0.75 

0.66 

Meta data, image categories 

Naive Bayes 

0.74 

0.74 

0.78 

Uni gram, 3-gram 

linearSVM 

0.85 

0.88 

0.84 

SVD + Unigram, 3-gram 

linearSVM 

0.85 

0.84 

0.88 

SVD +(Unigram, 3-gram), kernelPCA+(meta data, image categories) 

linearSVM 

0.87 

0.88 

0.87 


Table 2: Cyberbullying detection’s classifier performance 


of comments that are posted within one hour of one another 
in an Instagram media session. Fifth, we demonstrate that 
a Linear SVM classifier can significantly improve the accu¬ 
racy of identifying cyberbullying to 87% by incorporating 
multi-modal features from text, images, and meta data for 
the media session. 
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